System, method, and computer program product for performing fast, non-rigid registration for high dynamic range image stacks

ABSTRACT

A system, method, and computer program product are provided for performing fast, non-rigid registration for at least two images of a high-dynamic range image stack. The method includes the steps of generating a warped image based on a set of corresponding pixels, analyzing the warped image to detect unreliable pixels in the warped image, and generating a corrected pixel value for each unreliable pixel in the warped image. The set of corresponding pixels includes a plurality of pixels in a source image, each pixel in the plurality of pixels associated with a potential feature in the source image and paired with a corresponding pixel in a reference image that substantially matches the pixel in the source image.

FIELD OF THE INVENTION

The present invention relates to image processing, and more particularlyto techniques related to high dynamic range imaging.

BACKGROUND

Dynamic range in photography represents the ratio between lightintensity captured in the brightest part of the image and the darkestpart of the image. Film or an image sensor is exposed to light for a settime, known as the exposure time, and the film or image sensor reacts tothe amount of light that strikes the surface of the film or image sensorduring that time. Typically, the recording medium (i.e., film or pixelsites in the image sensor) has a limit to the range of intensities thatcan be recorded. For example, if the exposure time is short, therecording medium may not capture enough light in darker parts of theimage to record details about the object in the scene. In other words,part of the image is underexposed. Similarly, if the exposure time islong, the recording medium may capture too much light in brighter partsof the image such that details about the object are washed out. In otherwords, part of the image is overexposed.

High-dynamic range imaging uses multiple images of nearly the same scenecaptured using different exposure settings to create an image that has ahigher dynamic range than a single image captured using a singleexposure setting with the recording medium. For example, a series ofthree or more images may be captured using a digital camera in shortsuccession changing the exposure time and/or aperture size for eachimage. These images are then blended to increase the dynamic range ofthe composite image, enabling details to be shown in both darker areasand lighter areas of the same image.

Because the images in the high-dynamic range (HDR) image stack arecaptured at different times, objects in the image may have shifted. Inother words, both the camera could have shifted (e.g., if the camera ishand-held) or objects could be moving within the frame. Blending theimages without adjusting for this motion causes ghosting where objectsthat move appear translucent and are seen in two places at once.Existing techniques can be used to find a match for each pixel in areference image to a corresponding pixel in a second image. However,conventional techniques are either robust but slow (i.e., the techniquescan't be performed at interactive frame rates and may take minutes toregister a single set of HDR images) or fast but inaccurate (i.e., theblending still leaves visible image artifacts that are disturbing to aviewer). Some applications require HDR registration and blending to beperformed at interactive frame rates (such as when viewing HDR video inreal-time) while not sacrificing the quality of the product. Thus, thereis a need for addressing these issues and/or other issues associatedwith the prior art.

SUMMARY

A system, method, and computer program product are provided forperforming fast, non-rigid registration for at least two images of ahigh-dynamic range image stack. The method includes the steps ofgenerating a warped image based on a set of corresponding pixels,analyzing the warped image to detect unreliable pixels in the warpedimage, and generating a corrected pixel value for each unreliable pixelin the warped image. The set of corresponding pixels includes aplurality of pixels in a source image, each pixel in the plurality ofpixels associated with a potential feature in the source image andpaired with a corresponding pixel in a reference image thatsubstantially matches the pixel in the source image.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates a flow chart of a method for performing fast,non-rigid registration for HDR image stacks, in accordance with oneembodiment;

FIG. 2 illustrates a device configured to perform fast featurerecognition in two-dimensional images, in accordance with oneembodiment;

FIG. 3 illustrates the image processing pipeline of FIG. 2, inaccordance with one embodiment;

FIG. 4 illustrates two images in an HDR image stack, in accordance withone embodiment;

FIGS. 5A through 5F illustrate a conceptual diagram of a fast, non-rigidregistration (FNRR) algorithm for HDR image stacks, in accordance withone embodiment;

FIG. 6 illustrates a flow chart of a method for performing fast,non-rigid registration for HDR image stacks, in accordance with anotherembodiment; and

FIG. 7 illustrates an exemplary system in which the various architectureand/or functionality of the various previous embodiments may beimplemented.

DETAILED DESCRIPTION

The disclosed algorithm warps a source image by propagating an estimatedsparse flow field to a dense flow field and then correcting for errorsin the dense flow field. Known warp vectors at certain pixels may bepropagated to adjacent pixels in an edge-aware fashion such that objectsbounded by edges are warped in a substantially uniform fashion. Thetechnique results in a warped image with unreliable pixels that are thencorrected by blending pixel values from the source image with pixelvalues from a reference image.

FIG. 1 illustrates a flow chart of a method 100 for performing fast,non-rigid registration for HDR image stacks, in accordance with oneembodiment. At step 102, a warped image is generated based on a set ofcorresponding pixels. In the context of the present disclosure, the setof corresponding pixels comprises a plurality of pixels in a sourceimage, each pixel in the plurality of pixels associated with a potentialfeature in the source image and paired with a corresponding pixel in areference image that substantially matches the pixel in the sourceimage. Features may be referred to herein as edges and may be defined aspixels associated with peak gradients above a threshold value. In oneembodiment, the warped image is generated by propagating an estimatedsparse flow field fit to at least a portion of the pairs ofcorresponding pixels in the set of corresponding pixels to a dense flowfield in an edge-aware manner. It will be appreciated that the warpedimage is intended to be a modified version of the source image that isgeometrically consistent with the reference image except that theintensity of the pixels in the warped image reflect a difference in theexposure settings between the source image and the reference image.

At step 104, the warped image is analyzed to detect unreliable pixels inthe warped image. In one embodiment, gradients associated with pixels inthe warped image are compared to gradients associated with pixels in thereference image in order to detect whether pixels in the warped imageare reliable or unreliable. Pixels identified as reliable are assumed tobe accurately warped from the source image. Pixels identified asunreliable are assumed to be inaccurately warped from the source image.For example, pixels associated with objects undergoing non-rigid motionmay be unreliable. In addition, portions of the image without anymatched pixels within a bounded set of edges may be unreliable becausepropagating the dense flow is not continued across boundaries or edges.At step 106, a corrected pixel value is generated for each unreliablepixel in the warped image. In one embodiment, a patch match algorithm isimplemented to match a pixel location in the reference image thatcorresponds to the unreliable pixel to a pixel location in the sourceimage. The matching pixel in the source image is then blended with thepixel in the reference image to generate a corrected pixel value.

More illustrative information will now be set forth regarding variousoptional architectures and features with which the foregoing frameworkmay or may not be implemented, per the desires of the user. It should bestrongly noted that the following information is set forth forillustrative purposes and should not be construed as limiting in anymanner. Any of the following features may be optionally incorporatedwith or without the exclusion of other features described.

FIG. 2 illustrates a device 200 configured to perform fast featurerecognition in two-dimensional images, in accordance with oneembodiment. The device 200 may be, e.g., a desktop computer, a laptopcomputer, a tablet computer, a digital camera, a smart phone, a personaldigital assistant (PDA), or the like. As shown in FIG. 2, the device 200includes a system-on-chip (SoC) 210, a memory 204, a flash memory 242,and an image sensor 280. The memory 204 comprises one or more memorymodules for temporary storage of volatile data. In one embodiment, thememory 204 comprises one or more dynamic random access memory (DRAM)modules. The flash memory 242 comprises a flash memory device thatprovides non-volatile, long term storage of data. In some embodiments,the flash memory 242 may be replaced by other types of non-volatilememory such as read-only memory (ROM), solid state drives (SSDs),optical disks (e.g., CD-ROM, DVD-ROM, etc.), secure digital (SD) cards,and the like. It will be appreciated that data, as used herein, refersto both program instructions and raw data processed by the instructions.

The SoC 210 includes a central processing unit (CPU) 212, a graphicsprocessing unit (GPU) 214, a memory interface 230, a flash interface240, an image processing pipeline 250, and a system bus 220. Each of thecomponents of the SoC 210 may communicate with one or more of the othercomponents via the system bus 220. The SoC 210 is implemented on asingle silicon substrate and may be included in an integrated circuitpackage that provides an interface to a printed circuit board (PCB) thatincludes external interconnects to the other components of the device200. In one embodiment, the CPU 212 is a reduced instruction setcomputer (RISC) such as an ARM® Cortex A9, 32-bit multi-core processor.The CPU 212 may have one or more cores and may be multi-threaded toexecute two or more instruction per clock cycle in parallel. In otherembodiments, the CPU 212 may be a MIPS based microprocessor or othertype of RISC processor.

The CPU 212 retrieves data from the memory 204 via the memory interface230. In one embodiment, the memory interface 230 includes a cache fortemporary storage of data from the memory 204. The memory interface 230implements a 32-bit DDR (double data rate) DRAM interface that connectsto the memory 204. The CPU 212 may also retrieve data from the flashmemory 242 to be written into the memory 204 via the flash interface 240that, in one embodiment, implements an Open NAND Flash Interface (ONFI)specification, version 3.1. It will be appreciated that the flashinterface 240 may be replaced by other types of interfaces for flashmemory or other non-volatile memory devices, as required to interfacewith the particular type of non-volatile memory included in the device200. For example, the flash interface 240 could be replaced by an IDE(Integrated Drive Electronics) interface (i.e., Parallel ATA) forconnection to a solid state drive (SSD) in lieu of the flash memory 242.

In one embodiment, the SoC 210 includes a GPU 214 for processinggraphics data for display on a display device, such as a liquid crystaldisplay (LCD) device, not explicitly shown in FIG. 2. The GPU 214implemented in the SoC 210 may be a low-power version of a discrete GPU.The GPU 214 includes a plurality of streaming processors configured toefficiently process highly parallel tasks, such as pixel processing. TheGPU 214 may be configured to write processed pixel data to a framebuffer in the memory 204. A video interface, not explicitly shown inFIG. 2, may then be configured to read the pixel data from the framebuffer and generate video signals to transmit to the display deviceincluded in device 200 or connected to device 200 via an interface suchas an HDMI (High-Definition Multimedia Interface) connector.

In some embodiments, the device 200 also includes an image sensor 280for capturing digital images. The SoC 210 may transmit signals to theimage sensor 280 that cause the image sensor 280 to sample pixel siteson the image sensor 280 that indicate a level of a particular wavelengthor wavelengths of light focused on the pixel site. The level may beexpressed as a level of luminosity of a red, a green, or a blue channel,and the level is transmitted to the SoC 210 as raw image sensor data. Inone embodiment, the image sensor 280 is a CMOS (Complementary MetalOxide Semiconductor) image sensor. In another embodiment, the imagesensor 280 is a CCD (Charge Coupled Device) image sensor. It will beappreciated that the image sensor 280 may be included in an image sensorassembly that includes, in addition to the image sensor 280, one or moreof a lens, a shutter mechanism, a filter, and the like. Some imagesensor assemblies may include more than one lens, or the ability for auser to attach various lenses to the image sensor assembly that focuslight on the surface of the image sensor 280.

In one embodiment, raw image sensor data may be transmitted to the imageprocessing pipeline (IPP) 250 for processing. The SoC 210 may includeIPP 250 as a discrete hardware unit within the single silicon substrate.In another embodiment, the SoC 210 may implement the functions of theIPP 250 via instructions executed by the CPU 212, the GPU 214, or acombination of the CPU 212 and the GPU 214. The IPP 250 will bedescribed in more detail below in conjunction with FIG. 3.

FIG. 3 illustrates the image processing pipeline 250 of FIG. 2, inaccordance with one embodiment. As shown in FIG. 3, the IPP 250 includesan analog-to-digital converter (ADC) 310, a pre-processing engine 320, awhite balance engine 330, a demosaicing engine 340, a colortransformation engine 350, a gamma correction engine 360, a chromasubsampling engine 370, and a compression engine 380. It will beappreciated that the IPP 250 is shown for illustration purposes and thatthe particular stages of the IPP 250 implemented by variousmanufacturers may be different. In other words, the IPP 250 may includeadditional stages in addition to or in lieu of the stages shown in FIG.3.

In one embodiment, the ADC 310 receives the raw image sensor data and,for each pixel site, converts an analog signal into a digital value(i.e., an intensity value). In one embodiment, the ADC 310 has aresolution of eight or more bits and converts the analog signal for eachpixel site into an 8-bit intensity value between 0 and 255. In anotherembodiment, the ADC 310 is built into the image sensor assembly anddigital values are transmitted to the IPP 250 via a serial interface.

In one embodiment, the pre-processing engine 320 implements variousprocessing algorithms based on the raw image sensor data. In oneembodiment, the pre-processing engine 320 implements a filter to reducecross-talk between pixel sites. In another embodiment, thepre-processing engine 320 implements a noise reduction algorithm. In yetother embodiments, the pre-processing engine 320 implements an imagecropping algorithm. In still yet other embodiments, the pre-processingengine 320 implements an image scaling algorithm. It will be appreciatedthat various manufacturers of the device 200 may implement one or moreprocessing algorithms within the functionality of the pre-processingengine 320.

The white balance engine 330 is configured to adjust the intensityvalues for each color channel in the processed image data to account fora color temperature of a light source. For example, fluorescent lightingand natural sunlight cause the same colored object to appear differentin a digital image. The white balance engine 330 can adjust theintensity values for each pixel to account for differences in the lightsource.

The demosaicing engine 340 blends intensity values from different pixelsites of the image sensor 280 to generate pixel values associated withmultiple color channels in a digital image. Most conventional imagesensors include a color filter array such that each pixel site of theimage sensor is associated with a single color channel. For example, aBayer Filter Mosaic color filter includes two green filters, one redfilter, and one blue filter for every 2×2 array of pixel sites on theimage sensor. Each pixel site of the raw image sensor data is associatedwith only one color (e.g., red, green, or blue). The demosaicing engine340 applies a special kernel filter to sample a plurality of pixel sitesin the raw image sensor data to generate each composite pixel in thedigital image, where each composite pixel is associated with three ormore color channels (e.g., RGB, CMYK, etc.). The demosaicing engine 340decreases the spatial resolution of the digital image in order togenerate pixels of blended colors.

The color transformation engine 350 transforms the digital imagegenerated by the demosaicing engine 340 from a non-linear, devicedependent color space to a linear, device-independent color space. Forexample, the RGB color space is a non-linear, device dependent colorspace. The function of the color transformation engine 350 is to map theintensity of colors in the non-linear, device-dependent color spaceassociated with the image sensor 280 to a standard, linear color spacesuch as sRGB. The color transformation engine 350 transforms each pixelvalue (i.e., a vector of multiple color channels) by application of a3×3 color transformation matrix to generate a transformed pixel value.

The gamma correction engine 360 adjusts the intensity values of thepixels of the digital image such that the digital image, when displayedon a display device with a non-linear response, properly reproduces thetrue colors of the captured scene. The chroma subsampling engine 370divides the three chrominance channels (e.g., red, green, and blue) ofthe transformed pixels into a single luminance channel and two colordifference channels. Because human vision responds more to luminancedifferences than chrominance differences, the two color differencechannels can be stored using less bandwidth than the luminance channelwithout reducing the overall quality of the digital image. Thecompression engine 380 receives the uncompressed digital image from thechroma subsampling engine 370 and generates a compressed digital imagefor storage in a memory 204. In one embodiment, the compression engine380 compresses the image using a JPEG (Joint Pictures Expert Group)codec to generate a JPEG encoded digital image file.

It will be appreciated that the number and order of the variouscomponents of the IPP 250, set forth above, may be different in variousembodiments implemented by different manufacturers of the device 200.For example, in some embodiments, digital images may be stored in a RAWimage format and the demosaicing engine 340 is not included in the IPP250. In other embodiments, the chroma subsampling engine 370 and thecompression engine 380 are not included in the IPP 250 because thedigital image is stored in an uncompressed bitmap that describes pixelsin the sRGB color space. It will be appreciated that differentapplications require different combinations and order of enginesconfigured to implement various algorithms and that other processingengines, not described herein, may be added to or included in the IPP250 in lieu of the processing engines described above.

FIG. 4 illustrates two images in an HDR image stack, in accordance withone embodiment. The HDR image stack includes a plurality of imagesincluding at least a reference image 410 and a source image 420. Forexample, the HDR image stack may include five images captured withdifferent exposure settings, with the third captured image designated asthe reference image 410. In order to register each of the other imagesto the reference image 410, one of the other images in the HDR imagestack may be designated as the source image 420. Again, the referenceimage 410 may be captured with different exposure settings from thesource image 420. For example, as shown in FIG. 4, the reference image410 may be captured using a faster exposure time, as the scene appearsdarker than the source image 420. Furthermore, the scene captured in theHDR image stack includes both rigid and non-rigid motion. For example,the camera position may have moved slightly between the two imagescausing the background to shift. The shifting camera not only causes atranslation of stationary objects in the scene, but parallax causesdistant objects to move less relative to closer objects. The scene alsocaptures non-rigid motion of certain objects, as illustrated in theimages of FIG. 4 by noting that the subject's arms and ball are indifferent positions in the reference image 410 and the source image 420.

In order to create an HDR image from the reference image 410 and thesource image 420, the two images are blended. A naïve method of fusingthe images may be to simply blend each pixel in one image with acorresponding pixel in the other image at the same location relative toa particular corner of the images. However, with objects not located inthe same positions in the two images, such blending would createghosting that would be unpleasant to view. More specifically, the armsand ball of the subject would appear translucent in two different placesdue to the non-rigid motion of these objects. Thus, a more robusttechnique for blending images in the HDR image stack is needed toproduce higher quality composite images. It will be appreciated that,once each image in the HDR image stack is registered to the referenceimage 410, the HDR image may be created by blending three or more imagesin the HDR image stack.

FIGS. 5A through 5F illustrate a conceptual diagram of a fast, non-rigidregistration (FNRR) algorithm 500 for HDR image stacks, in accordancewith one embodiment. As shown in FIG. 5A, the FNRR algorithm 500includes a number of steps implemented by various blocks (i.e.,engines). As used herein, a block or engine is a set of logic, eithersoftware or hardware (or some combination thereof), that is configuredto implement a portion of the FNRR algorithm 500. The FNRR algorithm 500includes a feature matching & sparse optical flow (FMSOF) block 510, anincorrect match detection (IMD) block 520, a sparse to dense opticalflow (SDOF) block 530, a failure detection (FD) block 540, and a warpcorrection (WC) block 550.

The FMSOF block 510 selects two images from the HDR image stack. A firstimage is designated as a reference image 410 and a second image isdesignated as a source image 420. The FMSOF block 510 identifies anumber of pixels associated with features in the source image 420 andfinds matching pixels in the reference image 410 corresponding with eachof the identified pixels in the source image 420. The FMSOF block 510may find the set of corresponding pixels by any technique well-known inthe art. In one embodiment, the FMSOF block 510 may implement a sparseoptical flow algorithm such as the Lucas-Kanade algorithm in order tomatch pixels in the source image 420 with pixels in the reference image410. In other words, the FMSOF block 510 first identifies the pixels inthe source image 420 that are associated with features. For example, thesource image 420 may be divided into a plurality of blocks and eachblock of the source image 420 may be searched to discover a pixel in theblock having a maximum gradient. As long as the maximum gradient isabove a threshold value, then the pixel is identified as a featurepixel. It will be appreciated that any method may be implemented toidentify feature pixels in the source image 420. Once a plurality offeature pixels have been identified in the source image 420, then eachof the feature pixels are matched to a corresponding pixel in thereference image 410. The Lucas-Kanade algorithm provides one techniquefor finding a matching pixel in the reference image 410. First, a coarseestimate of a translation vector may be generated using low-resolutionversions of the images. Then, the coarse estimate of the translationvector may be refined using an iterative process based on the gradientsof the estimated matched pixel identified by the translation vector andan error between the identified pixel in the source image and theestimated matching pixel. The translation vector may be refined over anumber of iterations.

As shown in FIG. 5B, the FMS OF block 510 identifies a plurality offeature pixels in the source image 420 that are matched to correspondingfeature pixels in the reference image 410. The feature pixels and thematching feature pixels are circled in FIG. 5B. The output of the FMSOFblock 510 is a set of corresponding pixels that include the featurepixels in the source image 420 and corresponding feature pixels in thereference image 410. It is well-known that the Lucas-Kanade technique isrelatively accurate for well-textured images having small motionvectors. However, this technique may also introduce some mismatches ormatches related to feature pixels that are moving non-rigidly.

The IMD block 520 is configured to eliminate incorrect matches ormatches associated with non-rigid motion from the set of correspondingpixels. Again, the matching pixels identified by the FMSOF block 510 maybe unreliable for use in estimating a dense optical flow. In order topropagate a sparse flow field fit to the set of corresponding pixels togenerate a dense flow field for rigid motion, the set of correspondingpixels needs to be culled to remove outliers from the matching pixels.The matches may be incorrect matches (i.e., the pixels selected as amatch are not related) or the matches are correct but the objectassociated with the pixels is moving in a non-rigid manner. In oneembodiment, a RANSAC (RANdom SAmple Consensus) technique is used toidentify reliable matches in the set of corresponding pixels. In otherembodiments, other techniques may be used to separate the inliers fromthe outliers in the set of corresponding pixels.

In the RANSAC technique, the IMD block 520 selects a subset of thematched pixel pairs in the set of corresponding pixels. The subset ofmatched pixel pairs is used to generate an estimated homography matrixfor registering the source image 420 to the reference image 410 using,e.g., a least squares technique. The estimated homography matrix is thenapplied to all of the matched pixels in the set of corresponding pixelsto determine whether the estimated homography matrix is a good fit foreach particular pair of matching pixels (i.e., determining whether anerror between a warped pixel, generated by multiplying the pixel in thesource image 420 by the estimated homography matrix, and a correspondingpixel in the reference image 410 is below a threshold value). If theerror for a particular pair of matching pixels is small, then thosepixels are considered a reliable match. However, if the error for aparticular pair of matching pixels is large, then those pixels areconsidered an unreliable match. A score for the estimated homographymatrix may be generated that is based on the number of reliable matchesand/or unreliable matches identified in the set of corresponding pixels.The process is repeated a number of times for different subsets ofmatched pixel pairs using the same set of corresponding pixels. Then,the best estimated homography matrix is selected from all of theiterations and the reliable matches in the set of corresponding pixelsassociated with that particular estimated homography matrix may beremoved from the set of corresponding pixels and stored as reliablematches.

In one embodiment, the RANSAC technique may be reapplied to theremaining set of corresponding pixels. It will be appreciated that, eventhough the first iteration of the RANSAC technique likely removed thelargest set of matched pixel pairs fitting the estimated homographymatrix, the matched pixel pairs remaining in the set of correspondingpixels may also include other correctly matched pairs that move rigidlyand are, therefore, reliable matches for the purpose of propagating anestimated sparse flow field to a dense flow field. The RANSAC techniquemay be repeatedly applied to the remaining set of corresponding pixelsuntil the number of matched pixel pairs in the set of correspondingpixels is below a threshold value. When the number of remaining matchedpixel pairs is too low, then a subset of matched pixel pairs cannotrobustly support the generation of an estimated homography matrix. Theremaining matched pixel pairs in the set of corresponding pixelscomprise unreliable matches and may be removed from the set ofcorresponding pixels. Each of the matched pixel pairs previously removedfrom the set of corresponding pixels and identified as reliable matchesmay then be added back to the set of corresponding pixels to generate areliable set of corresponding pixels. As shown in FIG. 5C, the pixelsassociated with cyan circles are part of the reliable matches set andthe pixels associated with magenta circles are part of the unreliablematches set which have been removed from the set of correspondingpixels.

Once the IMD block 520 has refined the set of corresponding pixels, theSDOF block 530 warps the source image 420 to generate a warped image 535shown in FIG. 5D. The SDOF block 530 does not warp the source image 420using an estimated homography matrix based on the set of correspondingpixels because such homography matrices would not correctly warpportions of the image having non-rigid motion and would not compensatefor parallax. In one embodiment, the SDOF block 530 propagates the flowof pixels from the set of known matching pixels (i.e., a sparse opticalflow) to each of the other pixels in the source image 420. Thepropagation of the optical flow from the matching pixels to other pixelsmay be performed in an edge-aware fashion. In other words, flow vectorsfor a particular pixel in the source image 420 may be interpolated basedon nearby pixels included in the set of corresponding pixels so long asthere are no edges between the particular pixel and any of the nearbypixels used for interpolation. For example, large surfaces such as awall or whiteboard may be associated with pixels in the set of matchingpixels located at the corners of the surfaces. A flow vector for each ofthe pixels inside the surface may then be estimated based on the flowvectors of pixels at the corners of the surface because all such pixelsare inside the edges of the surface and move rigidly relative to saidcorners.

As shown in FIG. 5D, the resulting warped image 535 may be geometricallysimilar to the reference image 410 except the intensity of the pixels inthe warped image may be different. In other words, similar objects inboth the reference image 410 and the warped image 535 should appear inthe same location, but the colors and/or intensity of those objects maybe different based on the different exposure settings of the referenceimage 410 and the source image 420. It will be appreciated that theremay be errors introduced when the source image 420 is warped. As shownin FIG. 5D, the upper right corner of the image has sections where edgesprevented the optical flow to be propagated to these pixels. Inaddition, objects moving non-rigidly may be incorrectly warped and evenobjects that appear to be correctly warped, may be unreliable.

The FD block 540 is configured to determine which pixels in the warpedimage 535 are incorrect and which pixels in the warped image 535 arecorrect. In one embodiment, the FD block 540 may analyze the warpedimage 535 one scanline at a time. For each scanline, the FD block 540calculates a gradient for each of the pixels in the scanline. It will beappreciated that, for most images, the gradient values along thescanline will have peaks at edges (i.e., abrupt changes in luminanceand/or chrominance). These peaks may then be compared to a correspondingscanline in the reference image 410. These peaks should be aligned inboth the reference image 410 and the warped image 535. Pixels in betweenaligned peaks may be considered reliable in the warped image 535.However, pixels in between misaligned peaks may be considered unreliablein the warped image 535. A peak may be misaligned if the peak in thescanline of the warped image 535 is more than a threshold number ofpixels away from the corresponding peak in the scanline of the referenceimage 410. For example, if the peak is more than 2 pixels away from thecorrect pixel position in the warped image 535, then the peak isconsidered misaligned. It will be appreciated that gradient peaks may becalculated in only a single dimension (e.g., along the x-axis) thatcorresponds to the direction of the scanline. Pixels in the warped imagebetween misaligned peaks are considered unreliable. While the techniqueis described above with respect to scanlines in the horizontaldimension, the technique may be applied to columns of pixels as wellusing a gradient associated with the y-axis.

As shown in FIG. 5E, the FD block 540 may generate a pixel map 545 thatindicates whether each pixel in the warped image 535 is reliable orunreliable. The pixel map 545 may be a two dimensional array of 1-bitvalues that indicate whether a pixel is reliable/unreliable. The pixelmap 545 in FIG. 5E shows reliable pixels as black and unreliable pixelsas white. In one embodiment, a subset of the scanlines in the warpedimage 535 is analyzed and the pixels in the scanlines that are notanalyzed are marked as reliable/unreliable based on a correspondingpixel in an analyzed scanline. For example, only one scanline every nscanlines (e.g., 5 scanlines) is analyzed. Pixels in adjacent scanlinesto the analyzed scanline are then marked as reliable or unreliable basedon the corresponding pixel in the analyzed scanline (e.g., each pixel ina column of five adjacent scanlines is marked based on the single pixelin the column included in the analyzed scanline). This technique reducesthe amount of computations that must be performed. It will beappreciated that unreliable pixels in one scanline are likely to be nextto other unreliable pixels in an adjacent scanline. Such optimizationsmay reduce the number of computations performed without significantlyreducing the accuracy of the pixel map 545.

Once the FD block 540 has determined which pixels are reliable orunreliable in the warped image 535, the WC block 550 corrects thesepixels. In one embodiment, the WC block 550 is configured to implement acombination of dense flow estimation and a direct copy and paste fromthe reference image with blending. The dense flow estimation algorithmmay be any dense flow estimation algorithm well-known in the art. Forexample, in one embodiment, the WC block 550 implements a patch matchalgorithm that attempts to match a patch of pixels in the referenceimage 410 to a corresponding patch of pixels in the source image 420.

More specifically, for each pixel in the warped image 535 marked asunreliable, the patch match algorithm attempts to match a patch ofpixels in the reference image 410 corresponding to a location of theunreliable pixel in the warped image 535 to a patch of pixels in thesource image 420. Once the matching patch of pixels in the source image420 is located, the pixel at the center of the patch of pixels in thesource image 420 is copied into the unreliable pixel in the warped image535. The pixel in the warped image 535 may then be blended with thecorresponding pixel in the reference image 410 to produce a correctedpixel in the warped image 535. The process may then be repeated for eachof the unreliable pixels in the warped image 535 (i.e., for any pixelsmarked as unreliable in the pixel map 545).

In another embodiment, the WC block 550 may select pixels from thereference image 410 corresponding to locations for the unreliable pixelsin the warped image 535 and adjust the intensity of the selected pixelsto match an intensity of pixels in the source image 420 using anintensity mapping function. For example, the average intensity of pixelsin the source image 420 may be 20% brighter than the average intensityof pixels in the reference image 410. In this case, the WC block 550 maycopy a pixel from the reference image 410, increase the intensity of thepixel by 20%, and then blend that pixel with the unreliable pixel datain the warped image 535. It will be appreciated that the intensitymapping function may be a complex, non-linear, non-decreasing function.In one embodiment, the intensity mapping function may be estimated usinga large number of pixel correspondences. In another embodiment, theintensity mapping function may be estimated via a histogram mappingtechnique.

As shown in FIG. 5F, the WC block 550 generates a corrected warped image550 by combining the reliable pixels with corrected pixels. Theresulting corrected warped image 550 takes advantage of the lowercomputation complexity provided by sparse optical flow algorithms,propagates the sparse optical flow to each pixel in a similar manner tomore complex dense optical flow solutions, and corrects for any errorsin the resulting warped image. Such a technique may be orders ofmagnitude faster than applying a dense optical flow algorithm to thewhole image, while not being prone to the inaccuracies and imageartifacts associated with fast sparse optical flow algorithms that areapplied to the whole image.

It will be appreciated that the FNRR algorithm 500 may be implemented byany type of processor coupled to a memory storing the reference image410 and the source image 420. In one embodiment, the FNRR algorithm 500may be implemented, at least in part, by the GPU 214 of the device 200.In other embodiments, the FNRR algorithm 500 may be implemented by theCPU 212 or the IPP 250 of the device 200. In addition, the images in theHDR image stack may be captured by the device 200 using the image sensor280. In other embodiments, the images in the HDR image stack may betransmitted to a memory of the device 200, having been previouslycaptured by another external device.

FIG. 6 illustrates a flow chart of a method 600 for performing fast,non-rigid registration for HDR image stacks, in accordance with anotherembodiment. At step 602, the FMSOF block 510 is configured to detect oneor more pixels associated with features in a source image 420. In oneembodiment, the FMSOF block 510 detects pixels associated with featuresbased on at least one gradient value for the pixel. At step 604, theFMSOF block 510 generates a set of corresponding pixels that identifiesa matching pixel in a reference image for each of the one or more pixelsassociated with features in the source image 420. The set ofcorresponding pixels may be generated utilizing a Lucas-Kanadealgorithm. At step 606, the IMD block 520 refines the set ofcorresponding pixels to remove incorrect matches. The set ofcorresponding pixels may be refined using a RANSAC algorithm thatremoves outliers in the set of corresponding pixels. At step 608, theSDOF block 530 generates a warped image 535 based on the refined set ofcorresponding pixels. The SDOF block 530 may propagate the estimatedsparse flow field to a dense flow field in an edge-aware manner. Thedense flow field is used to generate the warped image 535.

At step 610, the FD block 540 analyzes the warped image 535 to detectunreliable pixels in the warped image 535. In one embodiment, the FDblock 540 is configured to select a scanline in the warped image 535 andcalculate a gradient value for each pixel in the scanline. The FD block540 then calculates a gradient value for each pixel in a correspondingscanline of the reference image 410. The position of peaks in thegradient values of the scanline in the warped image 535 and thecorresponding scanline in the reference image 410 are compared todetermine which pixels in the scanline of the warped image 535 arereliable or unreliable. The FD block 540 generates a pixel map 545 thatrepresents the reliable and unreliable pixels. At step 612, a WC block550 generates a corrected pixel value for each unreliable pixel in thewarped image 535. In one embodiment, the WC block 550 implements a patchmatching algorithm to select a pixel value from the source image 420that is associated with the unreliable pixel in the warped image 535.Then, the WC block 550 blends the selected pixel value with a pixelvalue from the reference image 410 that is associated with theunreliable pixel in the warped image 535 to generate a corrected pixelvalue for the unreliable pixel in the warped image 535.

FIG. 7 illustrates an exemplary system 700 in which the variousarchitecture and/or functionality of the various previous embodimentsmay be implemented. As shown, a system 700 is provided including atleast one central processor 801 that is connected to a communication bus702. The communication bus 702 may be implemented using any suitableprotocol, such as PCI (Peripheral Component Interconnect), PCI-Express,AGP (Accelerated Graphics Port), HyperTransport, or any other bus orpoint-to-point communication protocol(s). The system 700 also includes amain memory 704. Control logic (software) and data are stored in themain memory 704 which may take the form of random access memory (RAM).

The system 700 also includes input devices 712, a graphics processor706, and a display 708, i.e. a conventional CRT (cathode ray tube), LCD(liquid crystal display), LED (light emitting diode), plasma display orthe like. User input may be received from the input devices 712, e.g.,keyboard, mouse, touchpad, microphone, and the like. In one embodiment,the graphics processor 706 may include a plurality of shader modules, arasterization module, etc. Each of the foregoing modules may even besituated on a single semiconductor platform to form a graphicsprocessing unit (GPU).

In the present description, a single semiconductor platform may refer toa sole unitary semiconductor-based integrated circuit or chip. It shouldbe noted that the term single semiconductor platform may also refer tomulti-chip modules with increased connectivity which simulate on-chipoperation, and make substantial improvements over utilizing aconventional central processing unit (CPU) and bus implementation. Ofcourse, the various modules may also be situated separately or invarious combinations of semiconductor platforms per the desires of theuser.

The system 700 may also include a secondary storage 710. The secondarystorage 710 includes, for example, a hard disk drive and/or a removablestorage drive, representing a floppy disk drive, a magnetic tape drive,a compact disk drive, digital versatile disk (DVD) drive, recordingdevice, universal serial bus (USB) flash memory. The removable storagedrive reads from and/or writes to a removable storage unit in awell-known manner.

Computer programs, or computer control logic algorithms, may be storedin the main memory 704 and/or the secondary storage 710. Such computerprograms, when executed, enable the system 700 to perform variousfunctions. The memory 704, the storage 710, and/or any other storage arepossible examples of computer-readable media.

In one embodiment, the architecture and/or functionality of the variousprevious figures may be implemented in the context of the centralprocessor 701, the graphics processor 706, an integrated circuit (notshown) that is capable of at least a portion of the capabilities of boththe central processor 701 and the graphics processor 706, a chipset(i.e., a group of integrated circuits designed to work and sold as aunit for performing related functions, etc.), and/or any otherintegrated circuit for that matter.

Still yet, the architecture and/or functionality of the various previousfigures may be implemented in the context of a general computer system,a circuit board system, a game console system dedicated forentertainment purposes, an application-specific system, and/or any otherdesired system. For example, the system 700 may take the form of adesktop computer, laptop computer, server, workstation, game consoles,embedded system, and/or any other type of logic. Still yet, the system700 may take the form of various other devices including, but notlimited to a personal digital assistant (PDA) device, a mobile phonedevice, a television, etc.

Further, while not shown, the system 700 may be coupled to a network(e.g., a telecommunications network, local area network (LAN), wirelessnetwork, wide area network (WAN) such as the Internet, peer-to-peernetwork, cable network, or the like) for communication purposes.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method comprising: generating a warped imagebased on a set of corresponding pixels in a source image and a referenceimage; analyzing the warped image to detect a first subset of pixels inthe warped image; and for each pixel in the first subset of pixels inthe warped image, generating a corrected pixel value for the pixel,wherein analyzing the warped image to detect a first subset of pixels inthe warped image comprises, for each scanline in a plurality ofscanlines in the warped image: calculating gradient values correspondingto a plurality of pixels in the scanline, calculating gradient valuescorresponding to a plurality of pixels in a corresponding scanline inthe reference image, and comparing the gradient values corresponding tothe scanline in the warped image with the gradient values correspondingto the corresponding scanline in the reference image, and wherein thefirst subset of pixels includes pixels in the warped image identified asreliable based on a comparison of gradients associated with the pixelsin the first subset of pixels to gradients associated with correspondingpixels in the reference image.
 2. The method of claim 1, furthercomprising: identifying one or more pixels in the source imageassociated with features in the source image as feature pixels; andgenerating the set of corresponding pixels by matching the one or morefeature pixels in the source image with one or more corresponding pixelsin the reference image.
 3. The method of claim 2, wherein feature pixelsare identified by detecting pixels having gradients above a thresholdvalue.
 4. The method of claim 2, wherein the set of corresponding pixelsis generated using a Lucas-Kanade algorithm.
 5. The method of claim 2,further comprising refining the set of corresponding pixels based on aRANSAC algorithm.
 6. The method of claim 5, wherein the RANSAC algorithmcomprises: selecting a subset of corresponding pixels from the set ofcorresponding pixels; fitting an estimated homography matrix to thesubset of corresponding pixels; identifying one or more pairs ofreliable pixels from the set of corresponding pixels that fit theestimated homography matrix; and iterating the steps of selecting,fitting, and identifying a number of times.
 7. The method of claim 1,wherein the plurality of scanlines in the warped image comprises asubset of every scanline in the warped image.
 8. The method of claim 1,further comprising generating a pixel map that indicates whether eachpixel in the warped image is reliable or unreliable based on thecomparison of gradients associated with the pixels in the first subsetof pixels to gradients associated with corresponding pixels in thereference image.
 9. The method of claim 8, wherein the pixel mapcomprises a two-dimensional array of 1-bit values, each bitcorresponding to a pixel in the warped image.
 10. The method of claim 1,wherein generating the corrected pixel value for the pixel comprises:selecting a patch of pixels in the reference image based on a locationof the pixel; matching the patch of pixels in the reference image to acorresponding patch of pixels in the source image; and blending a pixelvalue associated with the patch of pixels in the source image with apixel value associated with the patch of pixels in the reference imageto generate the corrected pixel value.
 11. The method of claim 1,further comprising selecting the reference image and the source imagefrom a high-dynamic range image stack.
 12. The method of claim 11,wherein the reference image has an average intensity value that is lessthan or equal to an average intensity value of the source image.
 13. Anon-transitory computer-readable storage medium storing instructionsthat, when executed by a processor, cause the processor to perform stepscomprising: generating a warped image based on a set of correspondingpixels in a source image and a reference image in a high dynamic rangeimage stack; analyzing the warped image to detect a first subset ofpixels in the warped image; and for each pixel in the first subset ofpixels in the warped image, generating a corrected pixel value for thepixel, wherein analyzing the warped image to detect a first subset ofpixels in the warped image comprises, for each scanline in a pluralityof scanlines in the warped image: calculating gradient valuescorresponding to a plurality of pixels in the scanline, calculatinggradient values corresponding to a plurality of pixels in acorresponding scanline in the reference image, and comparing thegradient values corresponding to the scanline in the warped image withthe gradient values corresponding to the corresponding scanline in thereference image, and wherein the first subset of pixels includes pixelsin the warped image identified as reliable based on a comparison ofgradients associated with the pixels in the first subset of pixels togradients associated with corresponding pixels in the reference image.14. The non-transitory computer-readable storage medium of claim 13, thesteps further comprising: identifying one or more pixels in the sourceimage associated with features in the source image as feature pixels;and generating the set of corresponding pixels by matching the one ormore feature pixels in the source image with one or more correspondingpixels in the reference image.
 15. The non-transitory computer-readablestorage medium of claim 14, wherein the set of corresponding pixels isgenerated using a Lucas-Kanade algorithm.
 16. A system, comprising: amemory storing a reference image and a source image; and a processorconfigured to: generate a warped image based on a set of correspondingpixels in the source image and the reference image, analyze the warpedimage to detect a first subset of pixels in the warped image, and foreach pixel in the first subset of pixels in the warped image, generate acorrected pixel value for the pixel, wherein analyzing the warped imageto detect a first subset of pixels in the warped image comprises, foreach scanline in a plurality of scanlines in the warped image:calculating gradient values corresponding to a plurality of pixels inthe scanline, calculating gradient values corresponding to a pluralityof pixels in a corresponding scanline in the reference image, andcomparing the gradient values corresponding to the scanline in thewarped image with the gradient values corresponding to the correspondingscanline in the reference image, and wherein the first subset of pixelsincludes pixels in the warped image identified as reliable based on acomparison of gradients associated with the pixels in the first subsetof pixels to gradients associated with corresponding pixels in thereference image.
 17. The system of claim 16, the processor furtherconfigured to: identify one or more pixels in the source imageassociated with features in the source image as feature pixels; andgenerate the set of corresponding pixels by matching the one or morefeature pixels in the source image with one or more corresponding pixelsin the reference image.